Enable sccache for macOS CoreCLR legs - #128675
Conversation
…ch wrapper) Consolidate the -DCMAKE_C/CXX_COMPILER_LAUNCHER injection into eng/native/build-commons.sh so every native component invoked through build_native picks up sccache when USE_SCCACHE=true (CoreCLR, corehost, native libraries, etc.) instead of only the components that hand-rolled the flag themselves. On macOS, CMake wraps PCH includes in -Xarch_<arch> -include<path>, which sccache cannot parse and which also strips bare -include during preprocessing. Use a thin wrapper script that drops -Xarch_<arch> (safe in single-architecture builds) and rewrites -include<path> to -Xclang -include -Xclang <path>, which sccache passes through to the clang frontend correctly. Remove the now-redundant per-component sccache flag injection from src/coreclr/build-runtime.sh; build-commons.sh handles it for all build_native invocations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Widen the gate in setup-sccache.yml and sccache-stats.yml to also include osx + arm64 alongside linux + x64. The CoreCLR_Libraries leg in runtime.yml already lists osx_arm64 and references both templates, so widening the gate activates Mac sccache there without any change to runtime.yml. On linux/freebsd we keep using the linux-musl-x64 sccache NuGet package (also works for arm64 cross-compile on x64 hosts, per dotnet#128065). The NuGet package has no Mac binary, so on macOS we download the official tarball from the mozilla/sccache GitHub release matching the host CPU (uname -m, not archType: the sccache executable must match the agent's CPU). A pinned sha256 guards against tarball tampering and surfaces transient GitHub-side issues. --strip-components=1 places the binary directly in the tools directory so the existing prependpath line still finds it. Bump SCCACHE_CACHE_SIZE to 5120M to match dotnet#128065. The 3584M limit was evicting useful entries; the larger cache keeps hit rates consistent across the now-larger fleet of jobs. Starting narrow on osx_arm64 only: the xarch wrapper in eng/native/sccache-xarch-wrapper.sh assumes single-arch builds, so we widen to osx_x64 / additional Mac legs in follow-ups after validating server startup and cache hits in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Two changes together actually activate sccache on the x64 leg: 1. Widen the gate in setup-sccache.yml and sccache-stats.yml to also include (osGroup == osx && archType == x64). Without this, the templates short-circuit and do nothing on the x64 platform. 2. Reference both templates from the osx_x64 CoreCLR_Libraries job in runtime.yml (the one that builds for installer tests). That job previously had no sccache plumbing at all, so widening the gate alone would have been a no-op. The xarch wrapper script is target-arch agnostic — it strips any -Xarch_<arch> (matches both -Xarch_arm64 cross-builds and -Xarch_x86_64 native builds) and its only correctness precondition is single-arch builds, which build-commons.sh already enforces via single-valued CMAKE_OSX_ARCHITECTURES. The setup-sccache.yml binary picker uses uname -m to select the sccache executable, so an x64 host that cross-compiles arm64 still gets the correct x86_64-apple- darwin sccache binary while the cache key continues to differentiate target archs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wire setup-sccache.yml / sccache-stats.yml into the macOS legs in runtime.yml whose buildArgs are clean CoreCLR/libs/host invocations. Mono-flavored legs are deliberately excluded for this PR (see below). Legs added: - Libraries_CheckedCoreCLR (osx_arm64, osx_x64) - NativeAOT (osx_arm64) - NativeAOT_Libraries (osx_arm64) - Installer_Build_And_Test (osx_x64, osx_arm64) For Installer_Build_And_Test the new setup-sccache template is appended to the existing preBuildSteps (which downloads the CoreCLR_Libraries artifact); sccache-stats is inserted as the first postBuildStep so stats are emitted even if later steps fail. Mono legs (AllSubsets_Mono, AllSubsets_Mono_LLVMAOT, the *_RuntimeTests variants, Mono_MiniJIT_LibrariesTests) are intentionally NOT wired. src/mono/mono.proj invokes cmake directly via _MonoCMakeArgs and bypasses eng/native/build-commons.sh, so the launcher injection on this branch does not reach Mono's native build. Wiring sccache into those legs would download the binary and report a misleading hit rate dominated by Mono cache-misses without speeding the dominant work. Extending Mono itself to use sccache (by adding the launcher flags to _MonoCMakeArgs when USE_SCCACHE=true) is left as a follow-up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
github-actions
Bot
added
the
area-Infrastructure-coreclr
There was a problem hiding this comment.
Pull request overview
This PR wires up sccache for additional macOS CI legs by centralizing the compiler-launcher injection in eng/native/build-commons.sh, adding a macOS-specific argument-rewrite wrapper, and enabling the existing sccache setup/stats templates across the targeted pipeline jobs.
Changes:
- Move
CMAKE_*_COMPILER_LAUNCHERinjection behindUSE_SCCACHE=trueintoeng/native/build-commons.sh(and remove the older injection fromsrc/coreclr/build-runtime.sh). - Add
eng/native/sccache-xarch-wrapper.shto rewrite macOS-Xarch_*/-include...argument patterns before invokingsccache. - Extend pipeline templates and job definitions to download/configure sccache on macOS and print cache stats in the selected CI legs.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/coreclr/build-runtime.sh | Removes redundant sccache launcher injection now handled in shared native build logic. |
| eng/native/build-commons.sh | Adds centralized USE_SCCACHE handling and selects the macOS wrapper as the launcher on Apple targets. |
| eng/native/sccache-xarch-wrapper.sh | New wrapper script to rewrite Apple-specific compiler flags for sccache compatibility. |
| eng/pipelines/runtime.yml | Enables sccache setup + stats templates on additional macOS-related pipeline legs. |
| eng/pipelines/coreclr/templates/setup-sccache.yml | Adds macOS download path (GitHub release + SHA256 pin) and bumps cache size. |
| eng/pipelines/coreclr/templates/sccache-stats.yml | Expands stats collection gating to include macOS legs. |
| preBuildSteps: | ||
| - template: /eng/pipelines/coreclr/templates/setup-sccache.yml | ||
| postBuildSteps: | ||
| - template: /eng/pipelines/coreclr/templates/sccache-stats.yml |
| # sccache version. On linux/freebsd we use the linux-musl-x64 NuGet | ||
| # package; on macOS we download the matching tag from the mozilla/sccache | ||
| # GitHub release. Keep the version in sync across platforms so all | ||
| # jobs cache against the same compiler-launcher behavior. | ||
| sccacheVersion: '0.15.0' |
|
I ran builds on this PR twice. On the second run it shows sccache being used. Note the installer leg was still running when I collected this data.
|
|
@steveisok Awesome! Sorry, I merged and now you have merge conflicts :) |
|
Workflow state for the Holistic Review Orchestrator. {
"version": 5,
"last_dispatched_commit": "f89193586ddc6cd6f223ee6122f393868a911106",
"last_dispatched_base_ref": "main",
"last_dispatched_base_sha": "54cfeb453970e794043c71448a60a39b4fee3a38",
"last_reviewed_commit": "f89193586ddc6cd6f223ee6122f393868a911106",
"last_reviewed_base_ref": "main",
"last_reviewed_base_sha": "54cfeb453970e794043c71448a60a39b4fee3a38",
"last_recorded_worker_run_id": "29675342180",
"review_attempt_commit": "",
"review_attempt_base_ref": "",
"review_attempt_count": 0,
"max_review_attempts": 5,
"review_history_format": "holistic-review-disclosure-v1",
"review_history": [
{
"commit": "f89193586ddc6cd6f223ee6122f393868a911106",
"review_id": 4730526823
}
]
} |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Holistic Review
Motivation: Solid and real. This extends the existing sccache compile-caching rollout (#128065 for linux/freebsd) to 8 macOS CoreCLR/NativeAOT/Installer CI legs, cutting native rebuild times. macOS native builds are a well-known CI bottleneck, so the value is clear.
Approach: Reasonable and consistent with the prior rollout. The two pipeline templates are gated on osGroup==osx && archType∈(arm64,x64), so adding them to mixed-platform matrix entries is a no-op elsewhere. The sccache-xarch-wrapper.sh cleanly works around sccache's inability to parse Apple's -Xarch_<arch> PCH flags, and the SHA-256-pinned GitHub-release download for macOS is appropriate since no NuGet package exists for that host.
Summary: build-commons.sh always pins a single CMAKE_OSX_ARCHITECTURES, so stripping -Xarch_ is safe). My one substantive concern is scope: relocating the launcher injection from src/coreclr/build-runtime.sh into build_native() silently broadens sccache wrapping to corehost, native-libs, and Mono native builds on all platforms — including existing linux legs — which the PR frames as macOS-only. A maintainer should confirm that broadening is intended. No blocking bugs found.
Detailed Findings
⚠️ Scope — Launcher injection moved to shared build_native()
See the inline comment on eng/native/build-commons.sh. Removing the USE_SCCACHE block from build-runtime.sh and adding it to build_native() means every native build sourcing build-commons.sh (corehost, native libs, Mono) now gets the compiler launcher whenever USE_SCCACHE=true, on all platforms rather than only CoreCLR on the previously-enabled linux legs. Likely a positive expansion, but it should be an explicit, acknowledged decision rather than an implicit side effect of the refactor.
✅ Wrapper correctness — -Xarch_/-include rewriting
The wrapper strips -Xarch_<arch> and rewrites the following glued -include<path> into -Xclang -include -Xclang <path>. The safety argument is verified: lines 94/96, 147/149, 161, 173/175, and 187 of build-commons.sh each set a single-valued CMAKE_OSX_ARCHITECTURES, so there is exactly one target arch per compiler invocation and dropping the -Xarch_ selector is lossless. A non--include argument after -Xarch_ is preserved correctly by the else branch.
✅ Pipeline gating & cache isolation
The if or(and(linux...), and(osx, arm64|x64)) conditions in setup-sccache.yml and sccache-stats.yml correctly preserve the existing linux behavior while adding osx. The cache key already includes osGroup/osSubgroup/archType, so macOS entries stay isolated from linux ones. The 3584M→5120M size bump matches #128065.
💡 Download hardening
The macOS path uses set -euo pipefail, curl -fsSL, and shasum -a 256 -c - against a pinned digest — good. One nit: the pinned sha256 values will need updating in lockstep whenever sccacheVersion bumps; a comment already notes the version must stay in sync, which is sufficient.
Note
This review was generated by this repository's Holistic Review agentic workflow to complement the built-in Copilot review.
Generated by Holistic Review · 108 AIC · ⌖ 12 AIC · ⊞ 10K
| __sccacheLauncher="$__RepoRootDir/eng/native/sccache-xarch-wrapper.sh" | ||
| fi | ||
| cmakeArgs="-DCMAKE_C_COMPILER_LAUNCHER=$__sccacheLauncher -DCMAKE_CXX_COMPILER_LAUNCHER=$__sccacheLauncher $cmakeArgs" | ||
| fi |
There was a problem hiding this comment.
src/coreclr/build-runtime.sh (removed in this PR), so it applied to the CoreCLR runtime component build. By moving it into build_native() in build-commons.sh, the launcher is now injected for every native build that sources this file — src/native/corehost/build.sh, src/native/libs/build-native.sh, and Mono via src/tests/build.sh — whenever USE_SCCACHE=true is in the environment, on all platforms (not just macOS). On linux/freebsd legs where USE_SCCACHE is already set by setup-sccache.yml, corehost and native-libs compilations will now also be wrapped by sccache, which is a behavioral change not mentioned in the PR description. This is likely a beneficial expansion of caching, but please confirm it's intended and that the corehost/native-libs builds don't emit any flags sccache chokes on (the -Xarch_ handling is only applied on the osx/maccatalyst path). Worth calling out explicitly since the PR title/description scope this as a macOS-only change.
Resolve conflicts in setup-sccache.yml/sccache-stats.yml: main expanded the sccache gate to linux/freebsd/openbsd on x64+arm64; this branch added osx. Take the union of both conditions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 356ad667-b44e-4469-aadb-1527c3d1fe24
Use the sccache.osx-x64 package because both macOS target matrices run on x64 Azure Pipelines hosts. Restore the exact PackageReference for macOS because the package is unlisted and dotnet package download filters it out; keep the existing download path for the listed Unix package. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 356ad667-b44e-4469-aadb-1527c3d1fe24
Parameterize the existing dotnet package download command by package id instead of creating a temporary restore project. Select sccache.osx-x64 on macOS and specify nuget.org because the package is unlisted and is not present in the repository's default feeds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 356ad667-b44e-4469-aadb-1527c3d1fe24
There was a problem hiding this comment.
Copilot's findings
Comments suppressed due to low confidence (2)
eng/pipelines/coreclr/templates/setup-sccache.yml:47
- PR description doesn’t match the implementation here: it says macOS uses a GitHub release tarball with SHA-256 pinning and that these templates are gated on
osGroup == osx(no-op on non-macOS). In the actual template, macOS pullssccache.osx-x64via NuGet restore (no checksum verification), and the gate still includes linux/freebsd/openbsd as well as osx. Please reconcile by updating the PR description, or changing this template to match the described download+pinning and/or gating behavior.
# Download sccache and configure the environment.
#
# All supported platforms build on x64 hosts, including macOS arm64
# cross-builds. The unsuffixed package contains the linux-musl-x64 binary;
# macOS uses the osx-x64 package.
eng/native/sccache-xarch-wrapper.sh:6
- The header comment says sccache “drops plain -include flags during compilation”, but the script only rewrites
-include*when it immediately follows-Xarch_*(i.e., the CMake PCH pattern). This is likely just a stale/over-broad comment and could mislead future maintainers.
# Wrapper around sccache for macOS builds.
# sccache cannot parse -Xarch_<arch> flags that CMake generates for PCH
# includes, and also drops plain -include flags during compilation.
# We rewrite "-Xarch_<arch> -include<path>" to "-Xclang -include -Xclang <path>"
# which sccache passes through correctly to the clang frontend.
- Files reviewed: 6/6 changed files
- Comments generated: 0 new
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
eng/pipelines/runtime.yml:821
- These sccache pre/post templates are added to a platform matrix that includes
linux_x64andlinux_arm64in addition toosx_arm64. Because the templates are also enabled forosGroup == linux, this change expands sccache usage beyond macOS. If the intent is macOS-only, this matrix likely needs to be split (or the inclusion made conditional); otherwise, the PR description’s stated scope should be updated.
nameSuffix: NativeAOT
buildArgs: -s clr.aot+libs+tools.illink -c $(_BuildConfig) -rc $(_BuildConfig) -lc Release /p:RunAnalyzers=false
preBuildSteps:
- template: /eng/pipelines/coreclr/templates/setup-sccache.yml
postBuildSteps:
- template: /eng/pipelines/coreclr/templates/sccache-stats.yml
eng/pipelines/coreclr/templates/setup-sccache.yml:61
- The PR description says macOS sccache comes from a GitHub release tarball with SHA-256 pinning, but this template downloads
sccache.osx-x64from NuGet (and does not verify an external hash). Please reconcile the PR description with the implementation (either update the description, or switch this step to the tarball+hash approach).
if [[ "$osGroup" == "osx" ]]; then
sccachePackage="sccache.osx-x64"
sccacheSource=(--source https://api.nuget.org/v3/index.json)
fi
"$(Build.SourcesDirectory)/eng/common/dotnet.sh" package download "${sccachePackage}@${sccacheVersion}" -o "$(Build.SourcesDirectory)/.packages" -v quiet ${sccacheSource[@]+"${sccacheSource[@]}"}
sccacheDir="$(Build.SourcesDirectory)/.packages/${sccachePackage}/${sccacheVersion}/tools"
eng/pipelines/coreclr/templates/setup-sccache.yml:58
- This hardcodes the macOS sccache package to
sccache.osx-x64for allosGroup == osxbuilds. If the macOS pool ever moves these jobs to arm64 agents, the downloaded binary will be incompatible. Consider selecting the package based on the agent host architecture (or otherwise encoding/justifying the x64-host assumption).
if [[ "$osGroup" == "osx" ]]; then
sccachePackage="sccache.osx-x64"
sccacheSource=(--source https://api.nuget.org/v3/index.json)
fi
Bring up sccache compile caching for macOS CI legs in dotnet/runtime, mirroring the linux/freebsd rollout from #128065.
Scope
Wires sccache into 8 macOS CI legs that build native code through
eng/native/build-commons.sh:The two template files (
setup-sccache.yml,sccache-stats.yml) are gated onosGroup == osx && archType ∈ (arm64, x64), so adding them to mixed-platform matrix entries is a no-op on non-macOS platforms.How it works
eng/native/build-commons.sh— whenUSE_SCCACHE=trueis set in the environment,build_native()injects-DCMAKE_C_COMPILER_LAUNCHER=…and-DCMAKE_CXX_COMPILER_LAUNCHER=…into the CMake arguments. On linux/freebsd the launcher is baresccache; on macOS/maccatalyst it is the new wrapper described below.eng/native/sccache-xarch-wrapper.sh(new) — On macOS, CMake emits-Xarch_<arch> -include<path>for precompiled headers. Two issues for sccache:-Xarch_<arch>is an Apple-specific flag sccache doesn't recognize.-include<path>(glued together) prevents sccache from seeing the PCH path and hashing it correctly.The wrapper strips
-Xarch_<arch>and rewrites-include<path>into-Xclang -include -Xclang <path>beforeexec-ing sccache. Stripping-Xarch_<arch>is safe becausebuild-commons.shalways setsCMAKE_OSX_ARCHITECTURESto a single value (lines 94/96, 147/149, 161, 173/175, 187), so each compiler invocation has exactly one target arch.eng/pipelines/coreclr/templates/setup-sccache.yml— Downloads the sccache binary per-osGroup: NuGet for linux/freebsd (existing), GitHub release tarball for macOS with SHA-256 pinning. Tarball usestar --strip-components=1. Cache size bumped 3584M → 5120M (matching Enable sccache for arm64 and FreeBSD cross-builds #128065). Cache key includes${osGroup}${osSubgroup}|${archType}|${nameSuffix}|$(Build.BuildId)with rolling restoreKeys so per-platform/arch caches stay isolated.eng/pipelines/coreclr/templates/sccache-stats.yml— Same gate; runssccache --show-statsafter the build.Inspired by
#128065 (sccache for linux/freebsd cross-compile scenarios) — we adopt its cache-size bump and scope-widening pattern.